Game-Theoretic Procedures for Learning Structured Strategies from Demonstration

Acquiring new skills is an ability which would be very useful for a robot to possess. One conceivable method whereby a robot could learn new skills is if a teacher with expert knowledge of those skills could demonstrate these skills to the learning robot, in a variety of different settings. What is required would be that the robot does not merely learn to mimic the teacher exactly, as the teacher would be unable to demonstrate the skill in every possible state of the world. Instead the robot should be equipped with some robust method for adapting the learned skill to new situations. The research presented in this document embarks down a path towards solving this problem. Drawing on previous work from the fields of apprenticeship learning and reinforcement learning, an algorithm is presented for teaching an agent a new skill. This algorithm consists of two phases: the first in which the robot learns primitive skills from an expert, and the second which is a mechanism for composing these skills to provide strategies for successfully using the skill in unseen situations. This two phase solution is inspired by curriculum learning, and allows the agent to construct structured strategies in a hierarchical framework. The learning task is posed as a two-player game of the learning agent against disturbance processes generated by the environment or nature. By doing so, an equilibrium can be found using regret minimisation techniques to find an optimal mixture of the basis strategies into a composite strategy for a new scenario. A detailed example is provided, where the learning agent is required to learn how to navigate a busy multi-lane road of various shapes and sizes. The results demonstrate that the agent is able to successfully learn the primitive strategies from the expert. Then, in a variety of experiments, it is shown that the second phase of the algorithm can reconstruct parts of the basis that are present in new examples, as well as adapt the learned behaviours to totally unseen terrains, in a robust and flexible manner. By doing this, the agent shows an ability to synthesise its previously acquired knowledge in novel ways to overcome unpredicted challenges.